{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EAt-K2qgcIou"
   },
   "source": [
    "# Optimization Using Gradient Descent: Linear Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FZYK-0rin5x7"
   },
   "source": [
    "In this assignment, you will build a simple linear regression model to predict sales based on TV marketing expenses. You will investigate three different approaches to this problem. You will use `NumPy` and `Scikit-Learn` linear regression models, as well as construct and optimize the sum of squares cost function with gradient descent from scratch."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents\n",
    "\n",
    "- [ 1 - Open the Dataset and State the Problem](#1)\n",
    "  - [ Exercise 1](#ex01)\n",
    "- [ 2 - Linear Regression in Python with `NumPy` and `Scikit-Learn`](#2)\n",
    "  - [ 2.1 - Linear Regression with `NumPy`](#2.1)\n",
    "    - [ Exercise 2](#ex02)\n",
    "  - [ 2.2 - Linear Regression with `Scikit-Learn`](#2.2)\n",
    "    - [ Exercise 3](#ex03)\n",
    "    - [ Exercise 4](#ex04)\n",
    "- [ 3 - Linear Regression using Gradient Descent](#3)\n",
    "  - [ Exercise 5](#ex05)\n",
    "  - [ Exercise 6](#ex06)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Packages\n",
    "\n",
    "Load the required packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# A library for programmatic plot generation.\n",
    "import matplotlib.pyplot as plt\n",
    "# A library for data manipulation and analysis.\n",
    "import pandas as pd\n",
    "# LinearRegression from sklearn.\n",
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the unit tests defined for this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import w2_unittest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='1'></a>\n",
    "## 1 - Open the Dataset and State the Problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, you will build a linear regression model for a simple [Kaggle dataset](https://www.kaggle.com/code/devzohaib/simple-linear-regression/notebook), saved in a file `data/tvmarketing.csv`. The dataset has only two fields: TV marketing expenses (`TV`) and sales amount (`Sales`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex01'></a>\n",
    "### Exercise 1\n",
    "\n",
    "Use `pandas` function `pd.read_csv` to open the .csv file the from the `path`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "path = \"data/tvmarketing.csv\"\n",
    "\n",
    "### START CODE HERE ### (~ 1 line of code)\n",
    "adv = pd.read_csv(path)\n",
    "### END CODE HERE ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TV</th>\n",
       "      <th>Sales</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>230.1</td>\n",
       "      <td>22.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>44.5</td>\n",
       "      <td>10.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>17.2</td>\n",
       "      <td>9.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>151.5</td>\n",
       "      <td>18.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>180.8</td>\n",
       "      <td>12.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      TV  Sales\n",
       "0  230.1   22.1\n",
       "1   44.5   10.4\n",
       "2   17.2    9.3\n",
       "3  151.5   18.5\n",
       "4  180.8   12.9"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Print some part of the dataset.\n",
    "adv.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### __Expected Output__ \n",
    "\n",
    "```Python\n",
    "\tTV\tSales\n",
    "0\t230.1\t22.1\n",
    "1\t44.5\t10.4\n",
    "2\t17.2\t9.3\n",
    "3\t151.5\t18.5\n",
    "4\t180.8\t12.9\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "w2_unittest.test_load_data(adv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`pandas` has a function to make plots from the DataFrame fields. By default, matplotlib is used at the backend. Let's use it here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot: xlabel='TV', ylabel='Sales'>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmqElEQVR4nO2dfWwl13nen7Pc+2V+rFfSrbpRtKTduK3VINXuag0VNWQkJtN6hVryNg50UTRufQEtXK+hMImLLQyrKogYlY147QhBrhNQWDnI0k4cxzYQu7TpqlEctL7cXa4+TFWW2pKVVMW8jiVqV+YuudLpH3eGGl7Ox5mZMzNnZp4fMODlzJ2Z98zMfc6Z97znPUJKCUIIIeVhT9YGEEIISRcKPyGElAwKPyGElAwKPyGElAwKPyGElIy9WRugwg033CAnJiayNoMQQnLF+fPnfyylbA6uz4XwT0xM4Ny5c1mbQQghuUIIseq2nq4eQggpGRR+QggpGRR+QggpGRR+QggpGRR+QggpGRR+QgjRQK/Xw+LiInq9XtamBELhJ4SQmMzNzWF8fBxTU1MYHx/H3Nxc1ib5IvKQlvm2226TjOMnhJhIr9fD+Pg4NjY2ttc1Gg2srq6i2dw1dipVhBDnpZS3Da5ni58QQmKwsrKCarW6Y12lUsHKyko2BilA4SeEkBhMTExgc3Nzx7qtrS2YnGaGwk8IITFoNpuYnZ1Fo9HA2NgYGo0GZmdnM3fz+JGLXD2EEGIyrVYLk5OTWFlZwcTERKDo93o95e8mAVv8hBCigWaziaNHjwYKuQkRQIzqIYSQlEg7AohRPYQQkjGmRABR+AkhJCVMiQCi8BNCSEqYEgHEqB5CCEkYZxRP2AigJGCLnxCSa5JOjhb3+G5RPM4IoCySu1H4CSG5JenQyLjH7/V6aLfb2NjYwPr6OjY2NtBut7dFPqvQToZzEkJySdKhkTqOv7i4iKmpKayvr2+vGxsbw8LCAiYmJhIP7Uw9nFMIcbMQ4lEhxLIQ4gdCiPus9Q8IIV4UQly0lmNJ2UAIKS5Jh0bqOL5fFE+WoZ1JunquAfhNKeUtAG4H8FEhxC3WttNSylut5ZsJ2kAIKShJh0bqOL5fFE+WoZ2JCb+U8iUp5QXr8yUATwO4KanzEULMJKnOy6RDI3Udv9VqYXV1FQsLC1hdXUWr1UrFfj9S8fELISYAPAbg5wH8BoB/A+BVAOfQfyt42WWfewHcCwAHDx48srq6mridhBC9zM3Nod1uo1qtYnNzE7Ozs9vCp4ukE57l+fhePv7EhV8IMQLgLwH8tpTyq0KIGwH8GIAEMAPggJTyw37HYOcuIfnD5Jmp0iLrLJyZ5OoRQlQA/BmAP5ZSfhUApJQ/klK+LqV8A8AfAnhXkjYQQrLBlLw0WWFCFk4vkozqEQBmATwtpfysY/0Bx9c+AOCppGwghGSHKXlpVNHZFxEUv581Sbb4/ymAfw3glwZCNz8thHhSCPEEgF8EMJ2gDYSQjDAlL40Kulvnpr/tcAAXISRRsvZzB5FEX4Qp/RvMx08I0Yqqa0R1ZqqsSKJ1ruttJ6lQWAo/IQRAOJExueMyLEn1RXjF76uS6DWWUhq/HDlyRBJCkuPs2bOy0WjIffv2yUajIc+ePev53bW1NdloNCT6IdkSgGw0GnJtbS1Fi/Vil39sbCyw/Gmg6xoDOCddNJUtfkJKTtgIFNM7LqMQt3Wum6SvMSdiIaTk2CLj7Ii0RcbNJ523ME1Vms2mMf0QSV9jtvgJKTlhRSbrMM0sJi5Jm6SvMcM5CSHbOXUqlQq2traUcupkEaaZRu4fk4h7jTPL1aMDCj8hyVPGePui4yX89PETQgCY5eN2I2xfBPGGPn5CiC9hfepJ+eCL2qmcBRR+QognYQcRJTnoKOtO5SJBHz8hJcbPrx/Wp56WD970vgiTYK4eQsgOglrnYQcRuX1/aGhI+8Au03P/5AEKPyElRGW0blifutv3L1++jAsXLijblETfQBni/sNC4SekhKi05sP61JvNJk6fPr1r/fT0dKDoJtU3UKRkcjqhj5+QEhLGHx/Gp764uIj3vve9uHTp0va6sbExLCws4OjRo7FtCQPj/unjJ4Q4CNOaV/Gp2+6UkZERXLt2bce2oJDLqAnJglw4RUwmpwsKPyElRVdGSqc75ciRI2i326FCLqPE56u4cBj374NbrmbTFubjJ8RMvPLGLy8vy263q5Q/fm1tTc7MzMh6va6UDz9MrnrT8uynDTzy8TNlAyEkMl5pFC5fvuzp03fiTLomhMDHP/5xnDhxwvcNIUzqhlarhcnJydzE/ac1RoGuHkIKQFYhi3HcKW4hpZ/61Ke0nzMvcf9pRiBR+AnJOVmGLMZJoxC187WIqRvCzoIWF4ZzEpJjTAlZjOKiiGt7kVI3LC4uYmpqCuvr69vrgsJgVWA4JyEFxJSQxSjulLgtdxNcOLpcbGlHIFH4CckxboKxubmJl19+ORcpCsKGlJqUfkGni81ZCY6MjKBWq+H06dPJVWpuoT6mLQznJMQbZ8hipVKR1WpV7tu3r3Dhi3Y5TShbmJDSMHQ6HVmr1eTo6KiWMsIjnJM+fkIKQK/Xw9LSEu6+++7M/f1JYEpfhk0SPvkkykgfPyEFptlsYv/+/Ub4+5PAlL4MmyR88mmWkcJPSEEocoqCsGWL0xegsm8SIaWp3j83/49pC338hKhhYoqCtbU15fQNfqiWLU5fQNh9dZVt8Py67h/S9vELIW4G8EUAN6Lf+fEHUsrPCyGuA/BlABMAVgD8qpTyZb9j0cdPiDomxbc7UzJsbm5idnY2cjI4ILhscfzkpvQj6Lx/Wfj4rwH4TSnlLQBuB/BRIcQtAE4B+K6U8h0Avmv9TwjRhAnx7UAyo1GDyqbiJ/dy5ZjSj5DG/UtM+KWUL0kpL1ifLwF4GsBNAO4C8Ij1tUcA3J2UDYSQ7AgrpDpi9IP85H6x90XuI9mFm/9H94K+W+f/AhgD8IpjvXD+P7DPvQDOATh38ODBWH4uQkxHt6/YBKKkT9YRo+/lJ1exx8Q+kjjAw8efhuiPADgP4Lj1/ysD218OOgY7d0mRMWlgkk7W1tbkBz/4wR1Ce/LkSdfv6R4M5VaRdrtduW/fvh3nGRsbk91uN3DfvJKJ8AOoAJgH8BuOdc8AOGB9PgDgmaDjUPhJUUlqBGjWnD17Vtbr9R3l8iqbqiDHpajX2g8v4U/Mxy+EEABmATwtpfysY9M3AHzI+vwhAF9PygZCwpJ2LpisOhSTLKfdqXvlypVd29zKlpZvvYjpnCPjVhvoWAC8G/1a9QkAF63lGIDr0Y/meRbAAoDrgo7FFj9JgyxcLlm0QpMup1sLPqhsafrWi+TKCQJZ+fh1LBR+kjRZugH8OiN1C1SUcoa1w+0c9nmC5tL1O08UO8oi8F5Q+AnxIS0/sxeDIpVUqzxsOaPa4azM6vW6nJmZiSXAYe0oaod5WLyEn9k5CYE5ozaTtiXMsU2ZISusHSbdy6xhdk5CfIjb8aezszTJDt8w5Yxrh64RqGHtMGUErtG4vQaYttDVQ9Iiil9Yt1shjf4GlXIG2ZGWDz3s9Shj2KYXoI+fEP0kJTJZjCB1E3IvO9L2oYe9HkUbgRsVCj8hCZBkp3CaUSl+Qj5oR1Ytakb1hMdL+Nm5S0qHzrS3cdMAm5A+OWwZVKcdNKV8ZYadu4TAPztjFKJ2Cuu2Iw5hO0NVRtqaVD7igttrgGkLXT1EB0m6KMK4FUzrfIxij58P3bTylRmknauHENNIOkxSNXTRtHDDKG8trVYLq6urWFhYwOrq6o5ZtUwrn5O0czGZyt6sDSAkLUyZaMMUO5y0Wi1MTk6G8sk3m03X75lYPkD/NJB5hi1+UhpMyc5oih1udsUZcGW3pgEYV76o00AW9g3Bzf9j2kIfP9GJ7jC/qMdLO9wwyfO5hYOaFE4ZJey2CPl+wDh+QryJKlJZpXLOenTxoD2Dnbn1et0Iwbcp6+hfCj8hHkQVxbzk0k/azm63K0dHR3elYZ6ZmdFyfF2EGc2bdbZWXVD4CXEhjijGFQcdee5VbE1axDqdTqhJV7JE9ZoXvcXPzl1iLGl0rMUJPYwTvRJlgFNUW5OMsun1epiennbdZkoIpxPVDmxTO+C14VYbmLawxV8+0vKdx23ZRUkGFvWccWwNstOtJazSOo4yzWKeMKmDOgqgq4fkhbRfs+NmcgwrDnFcL3Fs9bLTrZJVrXijTrNI0oHCT3JDFh1rabbs4lZsOm31sqVeryvbp3uaRaIPL+HnyF1iHFmM/PQahZrUuWZnZ9Fut1GpVLC1tRXKf6zTVrvfwJmZc8+e3V1/tr/e7bxRRv2SbKHwE+OIK4x5wBSxdKtk33jjjb47wEFQxZtmxZkXTE5LTeEnRmIL49LSEgDg0KFDGVukHxPE0quSBVDoijdpTM8LxIlYiLHo+PGY3OoyCbfr5Hftsr6uWZ/fj6CJbdK03Wsilsw7blUWdu6WDx2RPUXItWIiWV/XTqcja7WaHB0dNfK++gUnpH3twKgekid0jIotwsjLuCSRkC7L6+o2Sti0++p1jZaXl1O/dl7Cz5G7xEjiRvbomgwkj2l5bZu/8IUvaJ/+MIlJVlSvca/Xw3333bdr/dDQkFEjhL1G/V6+fNmcCWrcagO/Bf0c/mNh94uzsMWfT+K2NuMOViqjq8i22S1pmo7Wpe4Wf5hr7JUMrlarGdXitxl8/rN4W0IcVw+AswDGAAwDWAbwAoCPq+yrY6Hw5w9dohml8rD36XQ6mVYcaeM1itZebFdZlhVykL1hUyUDkJ1OJ9L5s0DXtVMlrvBftP7+KwC/A6AC4ImAfR4GsAbgKce6BwC8COCitRxTOT+FP19kKZqDFU6n04kkcnlMy+uXN8e+B3ZlmEWFrGKv6uQoo6Ojslar5Ur0bdIcJR5X+H9gif2fAniPte7xgH3uAHDYRfh/S+WczoXCny+yEk2dFY7bsWq1mlxeXk7AcnWb/ATDq0X8lre8Rdbr9W3RN+UtJk6yujwmTsvCbi/hV+3c/QKAFfRdPY8JIcYBvOq3g5TyMQA/UTw+KRBZTbats+PR2UHXaDQA9FMZHDlyREsnaVhU0jgPdipWKhVUKhUIISCEQK/XM6dz0bL39OnTqNVqGBkZUU59HHdu4LRwdlpHScOdKG61gcoCYK/Cdyawu8W/AuAJ9F1B+1XOxRZ/OuhskaTty5QyGRfT8vKyrNVqmbaSo/jC5+fnYydfS5oiuG28cLoc6/W6rFarmVx3xHT13AhgFsC3rP9vAdBW2G9Q+G8EMIR+ZNBvA3jYZ997AZwDcO7gwYOJX6Cyk8Rk2Vm82uqucLzcVvPz86mVLYrrrNvtugr/zMxM6hWyG3nsPFclqJNd5f7pIq7wfwvAr8Ly66Of4+dJhf12CL/qtsGFLf5kcXtQK5VK7kIZbZJOW1ytVmW9XtdybVT89vPz86Fb6svLy66Cs7y8bISPPI+d56oEdbLnqcW/aP1dcqy7qLDfYIv/gOPzNIAvqZyfwp8sJj2oJjL4FlGpVLRcm6CQV+f2arUqK5WKckvdrcVfr9eNEdaytfjthlTab1pxhf+/AbgewAXr/9sB/GXAPnMAXgKwhX7cfxvAHwF4En0f/zecFYHfQuFPFpNeTaOSdCvWPv78/LyWlmqQ8Hltn5+fVypjHoQ1i36gtHArm0lRParCfxjAXwNYt/7+EMAvqOyrY6HwJ8/gLEppdUbp+DGkOcJWl6AGuTrcto+OjsozZ84onysPwmqC2ykpTChbLOHv74+9AP4RgJ8HUFHdT8dC4U8H54OahmjoEOwsWrY6rk2UFr8t/mHOaYL4kOzwEn7ffPxCiOOeGwFIKb/qt10XzMefDUnmDQ/KWa7K4uIipqamsL6+vr1ubGwMCwsLOHr0qFabnei4NvZ8A87JTpzzDdjb9+7di0uXLu3YN8q1IuXDKx9/0Axc/8JnmwSQivCTbEhyhii3uV795nX1IqvBYjquTdD0i/b2b37zm/jYxz62Q/yjXCtCbHyFX0r5b9MyhKSHCbMX6RLsvM/PG1SBNJtNHDt2DB/5yEd2rE+jciPFRTkfvxDiTiHEvxdC3G8vSRpGksGUoeNeOcujCHar1cLq6ioWFhawurqqZW5Tk/Lw67xWJuF1jU269oXFzfE/uADoAPgigOcB/Ef0QzJnVfbVsbBzVw+6O0J1dBya2Ploah5+E69VVLyusanXPq8gZjjnEwN/RwD8lcq+OhYKvx50jpY07QeqqxLyynFTBLE1BR1TExapEkwSL+FXdfXYPXA/FUL8DIBrAA5EfMkgGaHLr97r9dBut7GxsYH19XVsbGyg3W5n9mquw31lH+P48eM7OpyBbDNYFhGvLKrdblcpe6gp7spc41YbDC4APgngrQCOoz8a9yUAMyr76ljY4teHjhh0tzeH4eFhOT8/n4DF/ujI6R40ctn0Fn9arV9d54nT4s/DiGSTQBRXD4CjAP6u4/9fA/BtAL8L4Dq/fXUuFH696Mi66SaU9Xo9dZdPnFmcbDfVzMyMa66i4eFhI9xYg7gNtEva5ab7PF4NkKCGSVGTuyVVeUcV/gu2wKM/o9b/A/AvAcwA+IrfvjoXCn96qD6A9g8069axjnlb3fLU1+t15bw4Osqg+qMfFGBdCeOC7Euile1Vbr/rkWaLP603qSQr76jC/7jj8+8BeMDx/0W/fXUuFH59+D3MYR/A+fl5OTw8nHnrK4z7yqvFmFWe+jDXPKtkeqa0su1n155C0vR0IiokXZFFFf6nYM20BeB/ArjDuc1vX50LhV8Pfg9zlAcw6Yc2TItL9bt+NqcdKRL2+mWVPtsEv/rgs9vpdBK7V2mWN+lKNarwfwL9bJxfB7AEbOf2+TkAf+23r86Fwh+foIc56gOYVDK3JFtcKjanUQmEveZu99CeFCbpN5WzZ8/Ker0uh4eHlftykuwMTnLi+zTfcIxs8ff3w+0APgBg2LHu7wM4HLSvroXCH5+ghznOA6hbJNNocel0ecWxIWw5s8rzbp9XtcNb5zX0etOp1WqJ3Ju033CSzIQbWfhNWCj88VF5mE2Z/DpLn3IefvSmu6SSGCHu1beRlGsr7T4fo6J6TFko/HpQEZlOpyNrtVrovO868Yvzdv44kvixZFHpmD4KNew10XkNnR26tVot8c5s55tKvV6XMzMzxt4XFSj8BSdM56YJoXJBDFZSJ0+e3OE6GPxfVwWV9TWw789gJZfWeXU8F6oVdxCD7qLPfOYzu8RfdzBBnHtvYgVO4c8pKg+TLn+q2wTdjUYjs8ExThEMCmOMIixeZDVloX1eu6z256TPr/L8hL0mfhW3SkvaS4STDOWM86ZiWu4qGwp/DlF5mHS2UJeXl11FNanoCS8GKzvVMMZarabth2eCHz2NN44wz0/YaxJUcfvdJz8RTureRP0tZf2W6AeFP2eoPkxRWyluP55ut7vrVbper8du8Yf5obpVdioDl9IUyyTwq9yS7GNII++S25ukSiWThZhGedszZYCbGxT+nKH6MMUJCRxsHXc6He0CGndkqn1+L9fB2NiYrNVqu/YL88MzwTdrUovfrvB1uSu83iSD7lNWLrcobzZs8VP4tRDmYQrzA/HreHMTgDghnWF/ECpjDdyiesLkcR/EJN+sbYudNyhtH39SFU6UFr+NCZWyCllVUkFQ+HNIWEFX+YF4ieuZM2d2rR8dHY31uqpjZKqqv1n1Wjn3M7GlllVUT5J5l9J4qzABEyspCr8hOH/Y8/PzgRkgdT9MXmL3ve99T3uoXBw3lJ+Ae7XSg66VSjpmU3yzaZN0JTh4X/MeH58XKPwGMBiuZy/VajXVlo+Xv9y2q16va3td9RLyoLhx3WMN3Pazy2lSiz9LknZXmNgiLjoU/owJikxJW3D8/ONhEmANuk5U8qvH8atHjaAwLR2zqSwvL8szZ86kHsJLkoHCnzFBsejDw8OJxii7sba2Js+cOSNHR0cjuTucAl6tVmWlUgkU87ijOpOItU7ymueplWtSRzfRA4U/Y1Ra/PaoxDR+eM6EbFHePqK+wbhVgPV6PdTgq6guibCd5Sp9MCrny4OQmtjRTeJD4TeAwXA9p4/fFv00fnheoh0mMVvQG4zXW4PKYCzViidKS1plv7Nnz8pqtbptT6VSCS3aeRNSkwchkeikLvwAHgawBsdMXQCuA/AdAM9af/erHKsowi+ld1RPt9uN7HIJi9uPfGRkRJ45cybWoBVVkXO2vOMOvtKNX+ihV3m8RkHnSUjzVlERNbIQ/jsAHB4Q/k8DOGV9PgXgQZVjFUn4vXAbNVupVFJr8UfxdzsF3Pbxq7pfdAy+SoJut7srnt3ZBzOIX2ipSeVSwdRBSCQ6mbh6AEwMCP8zAA5Ynw8AeEblOEUX/rW1tV3unySFX0r3H3kUn7RKVE8UW7IiTIs/SNxNKpcqeeqMJsGYIvyvOD4L5/9+S9GFP2wr0w8/Ifb7P4kWatD5/b6bJao+fhV3jknlIuXDOOG3/n/ZZ997AZwDcO7gwYPJXRkNqMav++3v1uIPK7x+4ZVBE5fo9kkPvj28//3vl9VqNdOZvcKgEtWTR3cOKRemCH/hXD1e7pGwbpOTJ0/uEJC9e/eGEkeVaBk/gdIpYroid/JAHt05pDyYIvyfwc7O3U+rHCdL4Y+SPiBsh6VXOoEwwqgyWcmgG2kw57qOMEZVW0ZGRoyNcAkL3TnEVLyEfw8SQggxB+C/A/gHQogXhBBtAP8ZwJQQ4lkAk9b/xjI3N4fx8XFMTU1hfHwcc3NzO7YvLS1hz56dl7BSqaDb7aJare5av7Ky4nqelZWVXd+vVque33djYmICm5ubyt9/7bXXcPfdd+8o0+Tk5I7ybG1tod1uo9frKR9X1ZatrS1MTEyEOq6pNJtNHD16FM1mM2tTCFHDrTYwbcmixa8SseHll9fR4o/iCvELrzx58mRgP4JOP79Xjnd7iZPnnxCiBjhyNxxBc366iZqbjz/q5NRRfcV+UT1BOdd1d1aura3JmZkZWalUto83NDRE0SckJbyEX/S3mc1tt90mz507l+o5e70exsfHsbGxsb2u0WhgdXUVKysrmJqawvr6+o5tDz74IO65557tV/5er4eVlRVMTEwouQHCfl9nmezzzc3Nod1uo1KpYGtrC7Ozs5icnIxlV6/Xw9LSEgDg0KFDdIkQkhJCiPNSytt2bXCrDUxbsurctd05w8PDO2YL0pHrRhdhQka73e52TiC/NwvnMcPOmctOTkLMAXT1hMcWveHh4V2i59w2WAGkFaqoKsqD3+t0OtpTIOcpEyUhZYHCHxIV0et0Ojv813Dxm+u0Z3AUrIoox/Hbq3b2ciATIWbiJfyJhXPmHbcQS2dIZq/Xw/T0NLa2tnbtqztU0S2sNMg+1XL44RaW6Va2OOcghKQPhd+DINFzEzsAqNVqmJ2d1daB2ev10G63sbGxgfX1dWxsbKDdbmNzcxNXrlzxtE+1HH40m03Mzs6i0WhgbGwMjUbDtWxxzkEIyQC31wDTlqx9/CMjI7JWq+0IQ3Rzb4SZq1YVN3dLo9HYkce+0Wgo+fijhoqqTl7C1AWEmAXo449Gp9ORtVrNNWInDbFTyXujUuGkEXHDqB5CzMJL+BnH74NK3HvSsffAztj6q1evYs+ePTtsGhsbw8LCAo4ePZrI+Qkh+cQrjp8+fh/c/Ph79uzZHowERM/T0uv1sLi4qJQHp9VqYXV1FQsLCzvObUN/OiEkDBR+H9w6Ld2Sm4UlKPmbG3YF8853vlOpw5UQQrygqyeAubk5fPjDH94VQTPo8lFFxX2kepykXUyEkHzj5erZm4UxpuAUTwCuQtpqtXD99dfj+PHjeO2117bXO+PUnfsFCbLtPnIKv32sMALebDYp+ISQSJRW+O0O02q1ip/+9KcQQqDRaGBzcxOzs7NotVrb3z106BDeeOONHftvbW3hwoULeM973oNqtYrNzU20223Mzs5u/z94HIAx74QQA3AL9TFt0R3OGRQi6ZZuYDCs00525nUMr+NIyZh3Qkg6wCOcs5Qtfjd3i5NB18vc3Bymp6e3W/Kf//zncfjwYd9juB3HptVqxU51TAghUSml8AdNDeh0vThTJthMT0/j/PnzsaYXpI+eEJIVpQznbDabOH36NGq1GkZGRlCpVFCtVl3DI90SjUkpcfny5e2wytHRUdfznD59muJOCDGOUgq/03WztbWFhx56CC+88AIWFhawurq6o0N2ZGRklzvnypUrGBkZ2R5Y9dBDD+0S/5GRERw+fDiV8hBCSBhKJ/xO182lS5dw9epVTE9PA4DrCNznn3/e9Tj2+maziWPHjuHatWs7tr/++uuM1CGEGEnphD+J3PGq6YsJIcQESte5GzaO/tChQ9sTj9tUKhUcOnRox/cYqUMIyQula/FHaZ3ff//9qNVqGB4eRr1exyOPPOL6/agJ2wghJE1K1+IH1FvnztG9e/bswalTp3DixAkKOyEk15Q6SZtfXh1dydQIISQrmI9/AK/UyHae/KWlJU4gTggpJKV09ThDOu0Wfbvdxquvvrod33/16lXXxGwM0SSE5J3SCX+v18MnP/nJXYOy9u7di/vuuw9Xr17d3lapVNBoNLajehiiSQgpAqUSfruz1i2x2ubmJoQQO9YNDQ3ha1/7Gvbv388QTUJIYcjExy+EWBFCPCmEuCiESGVqLbdka05OnDixa5atK1eu4Oabb2aIJiGkUGTZufuLUspb3Xqck8BtxK5NvV7HnXfeiUajsWN9o9HA5cuX0zCPEEJSozRRPV6pmKvVKj73uc/tGonr3I8QQopEVsIvAXxbCHFeCHGv2xeEEPcKIc4JIc71er3YJxwcsVupVDA0NIRarYbp6WksLCww3w4hpBRkMoBLCHGTlPJFIcTfAfAdAB+TUj7m9X2dA7h6vR6WlpZw11137fDp24OzAPdJ1wkhJG8YNYBLSvmi9XcNwJ8DeFda5242m9i/fz9qtdqO9c5pEtmZSwgpMqkLvxBiWAgxan8G8MsAnkrThrAZOgkhpEhk0eK/EcD3hBCPA+gC+Asp5X9J0wDmzyeElBkmaaM/nxBSULx8/KUauTtIs9mk4BNCSkdp4vgJIYT0ofATQkjJoPATQkjJoPATQkjJKJXw27Nr6UgBQQgheaU0wu811SIhhJSNUsTxc+J0QkgZMSpXT9q45eLnxOmEkLJSCuFnbh5CCHmTUgg/c/MQQsiblCZlQ6vVwuTkJHPzEEJKT2mEH2BuHkIIAUri6iGEEPImFH5CCCkZFH5CCCkZFH5CCCkZFH5CCCkZhRZ+JmUjhJDdFFb4mZSNEELcKWSSNiZlI4SQkiVpY1I2QgjxppDCz6RshBDiTSGFn0nZCCHEm8Lm6mFSNkIIcaewwg8wKRshhLhRSFcPIYQQbyj8hBBSMij8hBBSMij8hBBSMij8hBBSMnKRskEI0QOwGmHXGwD8WLM5WVKk8hSpLECxylOksgDFKk/YsoxLKXeFNuZC+KMihDjnlqcirxSpPEUqC1Cs8hSpLECxyqOrLHT1EEJIyaDwE0JIySi68P9B1gZopkjlKVJZgGKVp0hlAYpVHi1lKbSPnxBCyG6K3uInhBAyAIWfEEJKRmGFXwjxz4UQzwghnhNCnMranrAIIVaEEE8KIS4KIc5Z664TQnxHCPGs9Xd/1nZ6IYR4WAixJoR4yrHO1X7R53ete/WEEOJwdpbvxqMsDwghXrTuz0UhxDHHtv9gleUZIcQ/y8Zqb4QQNwshHhVCLAshfiCEuM9an7v741OWXN4fIURdCNEVQjxulec/WevfJoT4vmX3l4UQVWt9zfr/OWv7hNKJpJSFWwAMAfhfAN4OoArgcQC3ZG1XyDKsALhhYN2nAZyyPp8C8GDWdvrYfweAwwCeCrIfwDEA3wIgANwO4PtZ269QlgcA/JbLd2+xnrcagLdZz+FQ1mUYsPEAgMPW51EAP7Tszt398SlLLu+PdY1HrM8VAN+3rvmfALjHWt8B8BHr878D0LE+3wPgyyrnKWqL/10AnpNS/m8p5SaALwG4K2ObdHAXgEesz48AuDs7U/yRUj4G4CcDq73svwvAF2Wf/wHgrUKIA6kYqoBHWby4C8CXpJRXpZT/B8Bz6D+PxiClfElKecH6fAnA0wBuQg7vj09ZvDD6/ljX+LL1b8VaJIBfAvAVa/3gvbHv2VcAvFcIIYLOU1ThvwnA847/X4D/w2AiEsC3hRDnhRD3WutulFK+ZH3+GwA3ZmNaZLzsz+v9Omm5Ph52uN1yVRbLNXAI/ZZlru/PQFmAnN4fIcSQEOIigDUA30H/reQVKeU16ytOm7fLY21fB3B90DmKKvxF4N1SysMA3gfgo0KIO5wbZf/dLrexuHm3H8DvA/h7AG4F8BKA38nUmggIIUYA/BmAX5dSvurclrf741KW3N4fKeXrUspbAfws+m8j/1D3OYoq/C8CuNnx/89a63KDlPJF6+8agD9H/wH4kf2Kbf1dy87CSHjZn7v7JaX8kfUDfQPAH+JNd0EuyiKEqKAvlH8spfyqtTqX98etLHm/PwAgpXwFwKMA/gn67jV7qlynzdvlsbbvA/C3QccuqvAvAniH1RNeRb/T4xsZ26SMEGJYCDFqfwbwywCeQr8MH7K+9iEAX8/Gwsh42f8NAL9mRY/cDmDd4XIwkgEf9wfQvz9Avyz3WNEWbwPwDgDdtO3zw/IBzwJ4Wkr5Wcem3N0fr7Lk9f4IIZpCiLdanxsAptDvt3gUwK9YXxu8N/Y9+xUA/9V6W/Mn617spBb0IxF+iL5/7BNZ2xPS9rejH3nwOIAf2Paj77v7LoBnASwAuC5rW33KMIf+K/YW+j7Jtpf96Ecy/J51r54EcFvW9iuU5Y8sW5+wfnwHHN//hFWWZwC8L2v7XcrzbvTdOE8AuGgtx/J4f3zKksv7A+AXACxZdj8F4H5r/dvRr6CeA/CnAGrW+rr1/3PW9rernIcpGwghpGQU1dVDCCHEAwo/IYSUDAo/IYSUDAo/IYSUDAo/IYSUDAo/IQoIIa53ZHr8G0fmRzmY4VEI8etCiN/PylZCgqDwE6KAlPJvpZS3yv5Q+g6A09bnE+gPEHRyD/qx/4QYCYWfkHh8BcCdjvzoEwB+BsBfZWkUIX5Q+AmJgZTyJ+iPmHyfteoeAH8iOTKSGAyFn5D4zOFNdw/dPMR4KPyExOfr6E+AcRjAW6SU57M2iBA/KPyExET2Z0x6FMDDYGuf5AAKPyF6mAPwj0HhJzmA2TkJIaRksMVPCCElg8JPCCElg8JPCCElg8JPCCElg8JPCCElg8JPCCElg8JPCCEl4/8DhE5wlJrCYZwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "adv.plot(x='TV', y='Sales', kind='scatter', c='black')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use this dataset to solve a simple problem with linear regression: given a TV marketing budget, predict sales."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2'></a>\n",
    "## 2 - Linear Regression in Python with `NumPy` and `Scikit-Learn`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the required field of the DataFrame into variables `X` and `Y`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "X = adv['TV']\n",
    "Y = adv['Sales']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2.1'></a>\n",
    "### 2.1 - Linear Regression with `NumPy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use the function `np.polyfit(x, y, deg)` to fit a polynomial of degree `deg` to points $(x, y)$, minimising the sum of squared errors. You can read more in the [documentation](https://numpy.org/doc/stable/reference/generated/numpy.polyfit.html). Taking `deg = 1` you can obtain the slope `m` and the intercept `b` of the linear regression line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear regression with NumPy. Slope: 0.04753664043301978. Intercept: 7.032593549127698\n"
     ]
    }
   ],
   "source": [
    "m_numpy, b_numpy = np.polyfit(X, Y, 1)\n",
    "\n",
    "print(f\"Linear regression with NumPy. Slope: {m_numpy}. Intercept: {b_numpy}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex02'></a>\n",
    "### Exercise 2\n",
    "\n",
    "Make predictions substituting the obtained slope and intercept coefficients into the equation $Y = mX + b$, given an array of $X$ values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# This is organised as a function only for grading purposes.\n",
    "def pred_numpy(m, b, X):\n",
    "    ### START CODE HERE ### (~ 1 line of code)\n",
    "    Y = m * X + b\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TV marketing expenses:\n",
      "[ 50 120 280]\n",
      "Predictions of sales using NumPy linear regression:\n",
      "[ 9.40942557 12.7369904  20.34285287]\n"
     ]
    }
   ],
   "source": [
    "X_pred = np.array([50, 120, 280])\n",
    "Y_pred_numpy = pred_numpy(m_numpy, b_numpy, X_pred)\n",
    "\n",
    "print(f\"TV marketing expenses:\\n{X_pred}\")\n",
    "print(f\"Predictions of sales using NumPy linear regression:\\n{Y_pred_numpy}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### __Expected Output__ \n",
    "\n",
    "```Python\n",
    "TV marketing expenses:\n",
    "[ 50 120 280]\n",
    "Predictions of sales using NumPy linear regression:\n",
    "[ 9.40942557 12.7369904  20.34285287]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "w2_unittest.test_pred_numpy(pred_numpy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='2.2'></a>\n",
    "### 2.2 - Linear Regression with `Scikit-Learn`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Scikit-Learn` is an open-source machine learning library that supports supervised and unsupervised learning. It also provides various tools for model fitting, data preprocessing, model selection, model evaluation, and many other utilities. `Scikit-learn` provides dozens of built-in machine learning algorithms and models, called **estimators**. Each estimator can be fitted to some data using its `fit` method. Full documentation can be found [here](https://scikit-learn.org/stable/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an estimator object for a linear regression model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "lr_sklearn = LinearRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The estimator can learn from data calling the `fit` function. However, trying to run the following code you will get an error, as the data needs to be reshaped into 2D array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of X array: (200,)\n",
      "Shape of Y array: (200,)\n",
      "Expected 2D array, got 1D array instead:\n",
      "array=[230.1  44.5  17.2 151.5 180.8   8.7  57.5 120.2   8.6 199.8  66.1 214.7\n",
      "  23.8  97.5 204.1 195.4  67.8 281.4  69.2 147.3 218.4 237.4  13.2 228.3\n",
      "  62.3 262.9 142.9 240.1 248.8  70.6 292.9 112.9  97.2 265.6  95.7 290.7\n",
      " 266.9  74.7  43.1 228.  202.5 177.  293.6 206.9  25.1 175.1  89.7 239.9\n",
      " 227.2  66.9 199.8 100.4 216.4 182.6 262.7 198.9   7.3 136.2 210.8 210.7\n",
      "  53.5 261.3 239.3 102.7 131.1  69.   31.5 139.3 237.4 216.8 199.1 109.8\n",
      "  26.8 129.4 213.4  16.9  27.5 120.5   5.4 116.   76.4 239.8  75.3  68.4\n",
      " 213.5 193.2  76.3 110.7  88.3 109.8 134.3  28.6 217.7 250.9 107.4 163.3\n",
      " 197.6 184.9 289.7 135.2 222.4 296.4 280.2 187.9 238.2 137.9  25.   90.4\n",
      "  13.1 255.4 225.8 241.7 175.7 209.6  78.2  75.1 139.2  76.4 125.7  19.4\n",
      " 141.3  18.8 224.  123.1 229.5  87.2   7.8  80.2 220.3  59.6   0.7 265.2\n",
      "   8.4 219.8  36.9  48.3  25.6 273.7  43.  184.9  73.4 193.7 220.5 104.6\n",
      "  96.2 140.3 240.1 243.2  38.   44.7 280.7 121.  197.6 171.3 187.8   4.1\n",
      "  93.9 149.8  11.7 131.7 172.5  85.7 188.4 163.5 117.2 234.5  17.9 206.8\n",
      " 215.4 284.3  50.  164.5  19.6 168.4 222.4 276.9 248.4 170.2 276.7 165.6\n",
      " 156.6 218.5  56.2 287.6 253.8 205.  139.5 191.1 286.   18.7  39.5  75.5\n",
      "  17.2 166.8 149.7  38.2  94.2 177.  283.6 232.1].\n",
      "Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.\n"
     ]
    }
   ],
   "source": [
    "print(f\"Shape of X array: {X.shape}\")\n",
    "print(f\"Shape of Y array: {Y.shape}\")\n",
    "\n",
    "try:\n",
    "    lr_sklearn.fit(X, Y)\n",
    "except ValueError as err:\n",
    "    print(err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can increase the dimension of the array by one with `reshape` function, or there is another another way to do it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of new X array: (200, 1)\n",
      "Shape of new Y array: (200, 1)\n"
     ]
    }
   ],
   "source": [
    "X_sklearn = X[:, np.newaxis]\n",
    "Y_sklearn = Y[:, np.newaxis]\n",
    "\n",
    "print(f\"Shape of new X array: {X_sklearn.shape}\")\n",
    "print(f\"Shape of new Y array: {Y_sklearn.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex03'></a>\n",
    "### Exercise 3\n",
    "\n",
    "Fit the linear regression model passing `X_sklearn` and `Y_sklearn` arrays into the function `lr_sklearn.fit`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### START CODE HERE ### (~ 1 line of code)\n",
    "lr_sklearn.fit(X_sklearn, Y_sklearn)\n",
    "### END CODE HERE ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linear regression using Scikit-Learn. Slope: [[0.04753664]]. Intercept: [7.03259355]\n"
     ]
    }
   ],
   "source": [
    "m_sklearn = lr_sklearn.coef_\n",
    "b_sklearn = lr_sklearn.intercept_\n",
    "\n",
    "print(f\"Linear regression using Scikit-Learn. Slope: {m_sklearn}. Intercept: {b_sklearn}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### __Expected Output__ \n",
    "\n",
    "```Python\n",
    "Linear regression using Scikit-Learn. Slope: [[0.04753664]]. Intercept: [7.03259355]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "w2_unittest.test_sklearn_fit(lr_sklearn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that you have got the same result as with the `NumPy` function `polyfit`. Now, to make predictions it is convenient to use `Scikit-Learn` function `predict`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex04'></a>\n",
    "### Exercise 4\n",
    "\n",
    "\n",
    "Increase the dimension of the $X$ array using the function `np.newaxis` (see an example above) and pass the result to the `lr_sklearn.predict` function to make predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "# This is organised as a function only for grading purposes.\n",
    "def pred_sklearn(X, lr_sklearn):\n",
    "    ### START CODE HERE ### (~ 2 lines of code)\n",
    "    X_2D = X[:, np.newaxis]\n",
    "    Y = lr_sklearn.predict(X_2D)\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TV marketing expenses:\n",
      "[ 50 120 280]\n",
      "Predictions of sales using Scikit_Learn linear regression:\n",
      "[[ 9.40942557 12.7369904  20.34285287]]\n"
     ]
    }
   ],
   "source": [
    "Y_pred_sklearn = pred_sklearn(X_pred, lr_sklearn)\n",
    "\n",
    "print(f\"TV marketing expenses:\\n{X_pred}\")\n",
    "print(f\"Predictions of sales using Scikit_Learn linear regression:\\n{Y_pred_sklearn.T}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### __Expected Output__ \n",
    "\n",
    "```Python\n",
    "TV marketing expenses:\n",
    "[ 50 120 280]\n",
    "Predictions of sales using Scikit_Learn linear regression:\n",
    "[[ 9.40942557 12.7369904  20.34285287]]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "w2_unittest.test_sklearn_predict(pred_sklearn, lr_sklearn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can plot the linear regression line and the predictions by running the following code. The regression line is red and the predicted points are blue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f80d9ab63a0>]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe4AAAE9CAYAAADNvYHXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/av/WaAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxbklEQVR4nO3dfZQb1X038O9dezdBxgm2TPzwtpLpk9CHkjaA25CmOCluG0hPSFrIUziicdvAhiVPgpvilkSnCZx0aRq3BZoA6eaYFJBiktD2QNo4rWkDTWlrajvgEHh4C9I+OCbGCwHCLrG9+j1/zGjRy8xoJN2ZuXfm+zlHZ3ellXR1Jc137svcUSICIiIissNI0gUgIiKi8BjcREREFmFwExERWYTBTUREZBEGNxERkUUY3ERERBZZmnQBwli1apUUi8Wki0FERBSLXbt2HRCRo71usyK4i8Uidu7cmXQxiIiIYqGUqvvdxq5yIiIiizC4iYiILMLgJiIisgiDm4iIyCIMbiIiIoswuImIiCzC4CYiIrIIg5uIiFKlWq2iWCxiZGQExWIR1Wo16SJpZcUCLERERGFUq1VMTExgbm4OAFCv1zExMQEAKJVKSRZNG7a4iYgoNcrl8mJoN83NzaFcLidUIv0Y3ERElBozMzN9XW8jBjcREaXG+Ph4X9fbiMFNRESpMTU1hVwu13ZdLpfD1NRUQiXSj8FNRESpUSqVMD09jUKhAKUUCoUCpqenUzMxDQCUiCRdhp7Wrl0rPK0nERFlhVJql4is9bqNLW4iIiKXDceA8zhuIiIi2HMMOFvcREREsOcYcAY3ERER7DkGnMFNREQEe44BZ3ATERHBnmPAGdxERESw5xhwBjcREWWO32FfpVIJtVoNjUYDtVrNuNAGGNxEROTDhGOaoyhD87Cver0OEVk87KvzsU14/Z5ExPjL6aefLkREFJ9KpSK5XE4ALF5yuZxUKhXry1AoFNoes3kpFAqRP3dYAHaKTyZyyVMiIupSLBZRr9e7ri8UCqjValaXYWRkBF7Zp5RCo9GI9LnDSmTJU6XUCUqpbymlHlZKfU8pdbl7/VVKqb1KqQfcy7ujKgMREQ3GhGOaoypDmMO+THj9fqIc4z4M4A9E5GQAZwD4sFLqZPe2a0XkLe7lGxGWgYiIBmDCMc1RlSHMYV8mvH4/kQW3iOwTkd3u7y8BeATAcVE9HxER6WPCMc1RlSHMYV8mvH5ffoPfOi8AigBmALwOwFUAagD2ALgZwIpe9+fkNCKi+FUqFSkUCqKUkkKhEOvENBPKkORzI8nJaUqpIwHcC2BKRP5OKbUawAE4s/Q+DeAYEfk9j/tNAJgAgPHx8dO9JgkQEaVdtVpFuVzGzMwMxsfHMTU1ZeSxxaRX0OS0SINbKTUK4B8A/JOI/KXH7UUA/yAipwQ9DmeVE1EWdZ5mEnC6a01czYv0SmpWuQKwBcAjraGtlDqm5d9+A8BDUZWBiMhmtpxmkuIV5azytwP4bQBndRz69Vml1HeVUnsA/DKA34+wDERE1jL5kCQbGbsSWp+WRvXAIvLvAJTHTTz8i4gohPHxcc9FQEw4JMk2ncMOzWVOAVg37MC1yomIDGX0IUmaxNUKTtOwA4ObiMhQtpxmclBhT/ahQ5qGHbhWORERJSLO9cCTXnu8X4nMKicionZpmRylS5yt4KiGHRJ5T/1WZjHpwpXTiChpw66ilfRpIk0U5vSaOuleCS3K9xQ8rScR0eB0LIRiW1dtHGxfYCbK9zSxldN0YXATUZJ0bKDDnAM6i2xe0jXK95TBTUQ0BB0baLa40yepFjcnpxER9aDj3MxZOCY7a5J6TxncREQ96NhA23BMNme99yep95Rd5UREIdg8FhuG7RPF0oZj3EREFIhj8GbhGDcREQVK05KgacfgJiIiLRPwKB4MbiIi4qx3izC4iYg0GWZWdtIzum2Y9U4OTk4jItJgmFnZnNFNnTg5jYioD4O0fsvlclvwAsDc3BzK5XKk96XsWZp0AYiITNLZ+q3X65iYmACAwNbvMLOy/f7H6/AsIra4iYhaDNr6HWZWtt//KKWsWb0s6TH6LGFwExG1GLTlPMys7KmpKSiluq4XkYG6y+MO0WYvRb1eh4gs9lIwvCPid6Juky6nn3760CclJyIKo1AoCICuS6FQ6HnfSqUihUJBlFJSKBSkUqmEfl6v5wQgSqm+yl+pVCSXy7U9Ri6X66ss/RqmzsgbgJ3ik4mcVU5E1CKpGd66lhxNYulSnmtcP84qJyIKKanjmXUtgKJj6dJ+u9q56lq8GNxERB1KpRJqtRoajQZqtVokod0ZjgC07DAMG6KDjFdz1bWY+fWhm3ThGDcRpUmU49DDPvag49XDjO9TN3CMm4jIHFGMQ7eeL3zlypUAgOeee67vc4dzvNoMHOMmIjKI7lNodnZvz87OYn5+HrfddlvfXf0cr/ZnyrHqDG4iopjpDkedS6ZyvNqbSceqM7iJKFVMaRUF0R2OOlvwPEuYN5PWk+cYNxGlhk1n2Wodk+53HLpTEsduZ03cY/9BY9wMbiJKjawGmE07LLYK/Gz9678CRx8NLF+u7fk4OY2IMkH3pC9bsHs7ep3DGz8NYLdSqNXrwE/9FPC+98VWFgY3EaVGlmdEx7FojKnimNdQKpWw9eqrsXtsDALgEQCntvZY//mfa39OPwxuIkoNr0lfSinU63VjJ6olzYbJfEEin+39/e8D69YBSuHcTZtw6sGDizd9LJ9H9bbbABHg1FP1PF8YfiuzmHThymlEFFZzBS+4Z9ZCjGfJsk0SZxLTLZIzk9XrIuvXiziR3Ha5dGwslvoCV04joqzJ6kS1fqShjrTN9t67F7j4YuCb3+y+7aabgA99CMU1a2KrL05OI6LMyepEtX6koY7CzmuoVoFiERgZcX5WqwD27QPOPRdQCjj++PbQvv56YGHBaWdfeimglDH1xeAmolTK8kS1sNJQR2EWs6lWgYkJoF53crheByYuehnVY/8A+PrXX73j5s3A4cPOP330o07KtzClvhjcRJRKaVm6M8rJY8PUke5yDfp4YQ6FK1+5gI5FzzCHZSjjGuCaa4BDh5ywvuIKYMkS3+cy5jPlN/g97AXACQC+BeBhAN8DcLl7/UoA2wE87v5c0euxODmNiAZh+6km45g8Nkgd6S5XJK/z+edFNmwQAURhwWuemSg1WFnj+EwhiclpSqljABwjIruVUssB7ALwPgC/A+A5EfmMUupKN7j/KOixODmNiLLI1Mljusul7fFefBH42MeALVvaHx9PoY6ix+MDps7BS2RymojsE5Hd7u8vwTle/TgA7wVwi/tvt8AJcyIi6mDKZKiwz+91fZgu8KFe549/DFx2mTPB7PWvbw/tTZuA+XlMVYro6OFGLgdYNmqyKJYxbqVUEcCpAHYAWC0i+9ybngGw2uc+E0qpnUqpnc8++2wcxSQiMsogk6HiWFAl/EzucIuj9P065+aAjRudsF6+3Dlcq2njRuDll53e8M9+Fnjta1EqAdPTTgtbKefn9DRg7eJyfn3oui4AjoTTTf6b7t8/6rj9+V6PwTFuovSxffw5DpOTk30tIhPXgiphnyfs4iihHm9+XmTTJs9FUeSyy0Reeknra0waAsa4ow7tUQD/BOBjLdc9CmfsGwCOAfBor8dhcBOlSxpW7IqaVx0ppWRyctL3PpGsIhZQvl47Xp07Ha2vI9TjvfKKyCc+4R3Wl1wi8sIL2l+XKYKCO8rJaQrOGPZzIrKx5frNAGbl1clpK0XkD4Mei5PTiNLF1ElXJmiep9urfoDgOor7nNG9DPQ+HzrkDD5ffXX3bRs2ANdeC6xYobegBkpq5bS3A/htAGcppR5wL+8G8BkAv6qUehzAr7h/E1GGmDrpKmmtY8J+gurIlAVCmkIf93z4sHM8tVLA2Fh7aF94IXDggNPO/pu/yURo9+TXFDfpwq5yosGYOo4cZ5duP5KuL796CVtHJg5B+Nbp4cMimzd7d4Off77I/v2JldkESGqMW9eFwU3UPxM34iaXzYQy+Y0J91OepHc+Ai0siFx3nXdYn3uuyL59SZfQGAxuogwytVXb5BcwSQXPMPWlq8xBLW7jQjisRkPkhhu8w/qcc0SefjrpEhqJwU2UQf3M6DVFkq3eQetLZ5njfv3D7nD43r/REPniF73Dev16kVotgleTLgxuogwyvcXtJckyD/rcusscV4/DsDsJXve/eGzMO6zXrRN58slIXkdaMbiJMsiEMdt+JdlLMGh92dizITL8Dkfz/hd4BTUgcsYZIo89Fu2LSLGg4OZpPYlSKszpDk2T5OFMg9aXaYdghTXUIXl/+7eo1esQAFtbrt4J4GTAie7//E/gjW8cvqDUzS/RTbqwxU2ULFu6b5NgY5lFBmhx33mnyOhoV8v6QUBOsWQoxiZgVzkRDcq2CVNJCFNm015XqPf1G98QyeW6u8FPOkn+8dOftnKHxRYMbiIamI2T3KIyaPia2ir3fD3bt4u8/vXdYb1mjciOHb3vT1oEBXdka5XrxLXKiZJj2vrXSWkuRzo3N7d4XS6XCzUObvza7PfeC7z//UDnKZRPOAG4/XbgF38xmXJlWFJrlRNRCtg6+Uq3crncFtoAMDc3h3K53PO+Rq7N/h//ARx/vLM++Dvf+Wpor14N3HOP086emWFoG4jBTWSharWKYrGIkZERFItFVKvVyJ4r9IkieoizzFEYJnz73fmJrK7uvx848UQnrN/+dmDvXuf6FSuA7dudsH7mGeAd79DzfBQNvz50ky4c4yZ6VRLjpTpW2DJxjLcfwy6JGvb1a6+rWk3kTW/qHrNetkxk27bBHpMiB05OI0oPGyeL2VjmTjpWGguz86OlrvbsETnllO6wHhsTueuu8I9DiQkKbk5OI7KMjZPFbCyzl2q1inK5jJmZGYyPj2Nqakr7gjYD19UjjwAXXQTs3t1+/ac+BaxfD5x5ptZyUrQ4OY0oRWycLGZjmb2USiXUajU0Gg3UarVIVqHrq64efxw44wxnzPrkk9tDe+tWoNEArroqkdC2fU6DyRjcRJbRNVksTjaWOSk96+r73wfWrXPC+k1vAnbsePUfb73VCWsR4IILnP9JQPPQuXq9DhFBvV7HxMQEw1sXvz50ky4c4yZqZ+PCFzaWOQ5e9dJ53d9dd51zOkyvk3ls2eKcRtMgOsbps/55ASenEdEgsrDxTPI1Bk54e/ppkbPP9g7rm24yLqxbDXvGtDQchTAsBjdRRugMoSxsPJN+jZ0t09WA3OkV1IDI9deLLCzEUq5h6Tpl6DAtdtsxuIkyQHcImbrx1LlzkvRrVErJ0YB8zS+sN28WOXw4lrLoNOxn0dZznOvE4CbKAN0hZOLGU/fOSWKv8cABeeptb/MM6ysBWQpY37MxzA5W0jtUJmBwE2WA7hDSufHU1UrWvUGPNSCef15kwwbPsP5jN6yzGlKdkh7CMAGDmygDdIeQro2nzo2w7p2TyAPihRdELr7YM6w/DciYx2sxoWfDBFmYGBmEwU0UIVM2MFGEkI7XpnOHIooWsvb376WXRCYnPcNaNm0SmZ/33QFhi5uaGNxEETGtS8+UnYhWOlvJpp1gpXlbDpAty5d7h/XGjSIvv9z2mH47ICZ8hsgMDG6iiHASTW9RdOHHtXMStKPw5Ztvlr9cutQ7rC+7zGl59/G4zR0cU3a4KFkMbqKImDjz2jSm9Ur0o3OnYwyQP/EKakD+GpDlfeyQmNg7QuYICm6eHYxoCMViEfV6vev6QqGAWq0Wf4EMFcdZtaIwMjKCJSIoA7jK4/a/AbARwAst19l2xjMyE88ORhQRnjwjnDjOqqXV4cPANdegIYJDaA/tKoC3HH88ioUCfhftoQ3Yd8azLEnLGcuWJl0AIps1A+jyyy/H7OwsAOCII45Iskg0qIUF4NprgU2bum76GoDLAByAs2M2/ZnPAAAmJiYwNze3+H/caTNX84xlzferecYyAObvSHZgi5tIg/n5+cXfZ2dntZzCMC2tA6M1GsD11zunv1y6tD20zz0X2LcP1UoFmwoFzCqFQqGA6elplEollEolTE9Po1AoQHXc1smW99KWcg6iXC637WQBwNzcHMrlMgDLXrvf4LdJF05OI5NFdWyxrRO6jNdoiNx4o/ds8HPOcc7KpZEt76Ut5RxU0ERSE187ODmNKDojIyPw+h4NM0mJk940EwG2bAEuuaT7tvXrgZtvBiIam7blvVy1atXicE8r08o5qKD3AYBx7xEnpxFFyG8y0jCTlGZmZvq6njyIALfe6nSDj4y0hfa9AM489lhUKxXg7rsjC23AjveyWq16hjZgVjmHETSR1Ib3qBWDm2hIUcwsj2JnALBsHG8QIsDtt78a1hs2LN7030uX4mdGR6EAvBPAv//gB1rmIvQS1XvpZdD3tznO6yUts+SD5iTE+R5p4deHbtKFY9xkOt2LaUS17rhp43ja3HGH55j1rpER+emE1wSPq96HeZ6gtdNT8fnowcTvBrhyGpF9K1XpLm/qlme9806R0dHuwH7zm0X27Om5HnjzEscqd3F89oZ5f/3um8/ntZfTVKZtH7QGN5zu9df1e79hLgxuGpaJe9Sdot5wpGJ51m3bRJYt6w7rk04S2b277V/DnIGrNdhM23D3a5j314bvR9YMHdwAvgzgdQCWAXgYwNMANvW4z80A9gN4qOW6qwDsBfCAe3l3mOdncNOwTG9txrHhNL0OfN19t8hRR3WH9YkniuzY4Xu3MC3uZh2nIbiGfX9t33FJGx3B/YD7swTgLwCMAtjT4z7rAJzmEdxXhHnO1guDm4ZlemszjlD1Cqdmd6hxG+l77hF5wxu6w/qEE0Tuuy/UQ3i93tHRUcnn813hZO1OTYs07HzEzeSdFR3B/T03rL8G4B3udQ+GuF+RwU0mMH3DHNeORaVSkXw+79vyTNR994kcd1x3WK9eLXLPPQNtZL3u43Wd6Tt2YZkcRKYxfUdHR3B/1O3i/gYABaAA4Nsh7ucV3DUAe+B0pa8I8/wMbhqW6V/SOHcsjNqJuf9+kTVrusN6xQqR7dsX/03X++f3OF47Mybt2PXCwO6ts45Mf8+HDm7POwJLQ/xPZ3CvBrAEzgS3KQA3B9x3AsBOADvHx8ejrSFKVFwbHZM3bnHuWCTeuty925lM1hnWy5Y5k8886NrZ8NtY5/N5o3fsgpi+U2oCv2Eik3tZdLS4VwPYAmCb+/fJAD4Y4n5twR32ts4LW9zp5bfRmZycNDZkoxLXjkVQCEZWhj17RE45pTusx8ZE7rqr59117GxUKpXAjbXJO3ZBjOpBMVSYiYqm1ZuO4N4G4H/DHdeGczrQ74a4X2eL+5iW338fwO1hnp/BnV5+X6jODTVbEPoE7Sxpbbk9/LDIqad2hfUCIOe17CiEoSOcgjbepmysB5F4D4oFwh4aaNJ2Rkdw/7f78zst1z3Q4z5bAewDcAjO4WMfBHAbgO/CGeO+qzXIgy4M7vQK+4WyfeNqGq/WpZaW22OPibz1rd0ta0C+/eEPS+6II0JvKFvLmM/nZXR0dKiNbFpXB2OLuze/Osrn88b2sugI7nsA5AHsdv8+A8C9Ye6r48LgTq9+urDYgojWwC23J58UOfNMz7CWW291TqMp/QWMV6/A2NiY56FcYQVtvG3GMe7ebKwjHcF9GoD7ALzg/nwMwM+Gua+OC4M7vby+UH4BksYWhEnjqn213Op1kbPO8g7rLVsWw7pVPzsGUbQibdx4h2XS58hUttXR0MHtPAaWAvgZAKcAGA17Px0XBne6dX6htI+1Gsq0IOlZnqefFjn7bO+wvukmz7Bu1U8YB3VrD7PhtW3jTdk1cHAD+M2gS9B9dV4Y3NljwgY26jKYODbZ+Zrv+PznRd7zHu+wvv56kYWFvh477I5KmCGUNO7METUNE9xfCrj4HoOt+8LgprjF0Ro2djbwD38oct553mG9ebPI4cMDP3TYnaGwx92mcfiESCQ4uJVzu9nWrl0rO3fuTLoYlCHFYhH1er3r+kKhgFqtZs1zhDY7C3zkI8DWrd23XXMNsGkTsHRprEWqVqsol8uYmZmB33ZKKYVGoxFruYjioJTaJSJrvW4b6eNBfl0p9YdKqU82L/qKSGSWmZmZvq4fxNTUFHK5XNt1uVwOU1NT2p4j0I9+BGzYACgFrFrVHtpXXw0cPOi0sz/+8dhDGwBKpRJqtRoajQYKhYLn/4yPj8dcKqLkhQpupdQXAPwWgI/AWav8/XDWKycKVK1WUSwWMTIygmKxiGq1mnSRQvELBJ1BUSqVMD09jUKhAKUUCoUCpqenUSqVtD1HlxdfBC6+2AnrFSuAW2999bZyGXjlFSesP/lJYHQ0unL0KfGdHCKT+PWht17gnsKz5eeRCHGSEV0XjnHbybRZ0/2wuexdXnpJZHLSe8x60yaR+fmkSxiKCRMWieICDcdx73B//heAYwG8FsATYe6r48LgtpOJs6b7YXVQvPyyyOWXe4b1w2ef7dxO1gvzGbX6c5xhOoL7jwEcBecwsH3u5dNh7qvjwuC2UxyzprlRajE/77SgPcL6c4Ass73ngNqE6RVKVc9Rxgwc3AB+HsD/aPn7AwD+GcBfAVgZdF+dFwa3naJucdu+UdKy0/HKKyKf+IRnWE8DctTIiNW9HuQvzPer3+8gd4TNMUxw724GNIB1AH4A4DwAnwZwR9B9dV4Y3HaKOlht7oofqm4OHhT51Kc8w/qJM8+UYzpO5BF1rwclI0yPVj+9XrbvCKfNMMH9YMvvNwC4quXvB4Luq/PC4LZXlHvwxi5gEkLfOx2HDolMTXmGtVx4ociBA4GPa+PODQXT3eK2eUc4jYYJ7ocALHV//78A1rXeFnRfnRcGN3kJCinTu/lC7XQcPuysVOYV1uefL1+78caunaIwp0m1vRXltTNoWhdvHOXRPcZt845wlJL6bA0T3GU4ZwO7E8B3gMWV1v4ngPuC7qvzwuAmL72WxTQ5oJydjgsFeEqABffnhVIcHxe57jrvsD73XJF9+0TEf4Ocz+c962LJkiXGhNow/E73Oey5uqMuY1Tl0Tmr3KQWtyk7YkkOHwwc3M59cQaA3wCwrOW6NwE4rdd9dV0Y3Ok2zJe0eV/buoQnJ78twI/bcjmHH0sFF7aH9TnnOGfl6uD3mvP5vBXjlIO+52GHApJ8700KwH6YMsZtSjlEkn0vhwpuEy4MbjuFbQ3o+JLa1s1XGG94NqoLeEpk/XrnfNcBgl6vKa0VP8O852GGApJ+7236LHqdUjfpz45JOz5JvpcMbopd2I2zri9pkl/20EHZaIjccosIIAoLnsEddntg0satX8OUnS1ufUxq2bYyaceHLW4Gd6aE/cDr+pImtRHq+byNhsjWrV0JXcBT3i3ugqbnNdgw7znHuPUxdXKnSTs+1o5xm3BhcNsn7MZZ55c0iS5iv/J/aNWq7lQGRE4/XeThh6VSEcnl2m/K5UT6KXI/57ZOuvuz1bDvuQ2zyicnJ2XJkiUCOBMDJycn+7p/HK+n17BDUjsbpu34WDer3JQLg9s+YTfOpn1J+9W68XsPID/xCus3v1lkz56u+1YqTgtbKednFC/ZxPo1sUw6Dfv64qqfMMMOSXXvm7YjlgQGN8Wun41P68zwZivFli/rB97wBnnJI6yfWLpUZNeupItnVLdjqzRvmIet87jes16HUw4yZDVsedL6mRgEg5sS0c8X0apW2N13ixx1VHdYA/LzcMZc8/l81+tOYsNk0kSfrBi2zqN8z/xmkSfd4rbq+x8TBjcNJY7AMbVluOiee0SOProrrOWEE+Sbn/zkYv3k83nPiVKTk5OJbJiMr9cUMrXFHRSOSQen7techtY7g5sGFtcXOqi7LjH33Sdy3HHdYb16tRPkHvw2QM0hgLgDNOkNchitG9l8Pu/ZW2FK+cKUKcox7mECqVc4Jhl2OnsZbPjMh8Hgpjb9fEHjarH5BduSJUu0Pk9PO3aIrFnTHdYrVohs397z7v0sENJal1GvaW1q66PXOGvSG9xBQ2DYOvebOe9VV/l8fqjFa0wYNtG5nUlLLxODmxb1uyGK68ueaIt7926Rk07qDutly0S2bfO8i9+Gud8Wd2f9Jh1UcQsaXzVhg2tSCATVVZjPjUmvpZPOVrLJOyj9YHDTon6/vHF92ePeqPzDn/6pPDI62h3WY2Mid90VeN9Bxgq9xrj9NjAmbEjjEqaHIskNblD54u69GHYHx/QuZF09QybvoPSDwU2L+t0b1fFlD/OFnJycjL71+fDDIqee2hXWC4BcODYW+rkGHSvsvN7EoIqbrS3uJILPr9emn8+NycMmupi+gxIWg5sWDbI3OsyXPcyXyOt/lFJ9rzbl6dFHRd761u6WNSC/NWBA6OqK090ysHGjbOMYd1I7Fibv4JjGxu9CJwY3LYp7bzRMOGnv2nrySZEzz/QMa7n1VlFDtnR1lVfne2FzK8OWWeVJ95AMO8ZNdmFwU5s490bDtE61tGDrdZGzzvIO6y1bnJN9uIYN3n5DMqi+BznUyOv/0zKuZ7Kk63jYWeVkFwZ3CtnSFRS0sevVkum5QXz6aZF3vcs7rG+6qS2sW8U1bq/rucI8Vlpm0prMhF4NW773NDwGd8p4TeQydc+7n1nWoTaIP/iByHve4x3W118vsrAQulxxbAB1ttKCHivp1mBWMDgpLgzuFAlqXZk61uW1sevV0m57DT/8och553mH9ebNIocPR1rWYehsCQc9lgmtQSLSh8GdIkGBZ1Mrq2egHTggcuGF3mF9zTUihw61PZ6OCU5RhF9cLe5m+dkajBbrmOLC4E4R0xesCNK60fM6JvUoQO5Ytsw7rK++WuTgQd/H1XFIURTdzVmYOZ6VMDO1/imdGNwp0k+L26QNql+4Lgfki15BDYiUyyKvvNLzsXX1QvTqih60LnW+Dya9p83yZCXMOI+A4sTgTpGwrUvTNqitG71lgNzgF9abNonMz/f12Lp6Ifw2zPl83qi6HESlUpF8Pt/2mnSUP0thxpn7FCcGd8qEGc81bYOaA+Rav7DeuFHk5ZcHfmxdLW6/nZ3WwDOhLvtVqVRkbGysq/yjo6NGTb4znWnfKUq3RIIbwM0A9gN4qOW6lQC2A3jc/bkizGMxuPtTqVQCQyw28/MiV1zhGdafc1veOjZ6OpfN9OqKtj2ces3gD9Kraz5LYWZaLxalW1LBvQ7AaR3B/VkAV7q/Xwngz8I8FoM7vF4h1hyvjcwrr4h84hOeYb1lyRJZHtFGL8plM20Pp6ChhKCdj0HXmU9zmJk2x4DSK7GucgDFjuB+FMAx7u/HAHg0zOMwuMNL5HCxgwdFPvUpz7CWDRtEnn9eRPQu/Rkn28Np0BZ32B0Wk987IluZFNw/avldtf4ddGFwhxfb4WKHDolMTXmH9YUXOsdhh2RDMNocToOOcds+REBkMyOD2/37+YD7TgDYCWDn+Ph4ZJWTNpG2uA8fll0XXOAZ1l8F5NTjj18Mgn6CLs6u6M5yTU5OWhvI/RhkVrntQwRENjMpuNlVHrFKpSKjo6O+od13S3ZhQeS66zzD+u8BWe3x+F7rkAc9b1wtu17j/2FaoVliQ08IUVqZFNyb0T457bNhHidtwR2mNTpo16xft2izpdQ+rixSKIgo5fxcvGlhQeSGGzzD+h8BObZHi95rVbSgllpcLbswvRHN1ig5bB4iILJZIsENYCuAfQAOAXgawAcB5AH8C5zDwe4GsDLMY6UpuKOeqRt+QpFILteey7mxg1KBx/rg69eL1Gqhxs+DLn4taK/Xq5SSycnJoeq6Uz/lJyJKUmItbl0XG4I7bMskTLAO0wIN2+1cKHg2qKWAp5xf1q0TefLJUGUP2+L2avU3eZ2qVHe3bNjyM7iJKGkM7oiFbSEHLYzSGqzDjPn2DP1GQ+TLXxaFBc/gVqrR1+vsvAx6ru04usvDlB9gVzkRJY/BHbEwodMrNHS1uP12Iu796EfbErqAp7xb3D2eIuys7DDn3G4V5wS1oB6BsbExjuMSUeIY3BELEzpBIaZ7NapmaJ4LyE+80vnNb5bKn9a7x7hzLRPUYqwbkfgPCfPaidJ14g0iomExuCMWJnR6TYzqnvE94Gzeb3yje9YZIHLSSSK7drX9q++sco3CT5bz3lmJ6jhrzpYmIpMxuCMWZmJVmIlRA0/GuvtukaOO6g7rNWtEduwY+HXpCLd+eg+8uuEH7XlgMBORzRjcEQp7KFPYiVGhu4bvuUfk6KO7w/qEE0Tuuy+S1zVoaA7aah60+5wLhxCR7RjcEeonXHpN2Oo5Geu++0SOO647rFevdoI8odfV+Rp1heagE9biHC8nIopCUHCPgIYyMzMT+vpSqYSpqSnkcjnfxxsfH2+/4v77gRNPBJQC3v52YO9e5/oVK4Dt253ofuYZ4B3v6Lvs1WoVxWIRIyMjKBaLqFarA72uVuVyGXNzc23Xzc3NoVwu912+rrrocX3ToGUnIrIBg3tI/YaLV7A15XI5TE1NAd/5DnDSSU5Yv/WtwFNPOf+wbBmwbZsT1s89B/zKrwxc7mq1iomJCdTrdYgI6vU6JiYmFsPbhND02slZrKMAg5adiMgKfk1xky4md5X32zXs1/17CiDPH398dzf42JjIXXdpL3ev7uRBu7x1d1MPMsmMY9xEZDtwjDtag57C8n8BsqszqAHn+Kw77oi0zEGHpzVfSz6fl3w+b2VoclY5EdmMwR2TMGFx5+bNsmNkpDusAZGtW50lSWPg1zLWsV44Q5OIaDhBwa2c2822du1a2blzZ9LFCNQcM24dv87lcpienkbpbW8Dfud3gG9/u+t+H8vncfp116F00UUxlta7vEopeH0eCoUCarVajKUjIso2pdQuEVnreRuDW49isYh6vb749ziAmwGs9/rnLVuA3/1dZ/JZgqrVKsrlMmZmZjA+Pt5W/lZKKTQajZhLR0SUXUHBzVnlmszMzOA4ANvg9DHX0RHaN90ENBpOp/jv/V7ioQ04h6fVajU0Gg3UajUUCgXP/+NsbCIiczC4h7VvH3DuuWiI4GkAZ7fc9FEAoyMjqFYqwKWXag/roOOwBzHo4VdERBQfBvcg9u8Hzj/fCeJjjwW+/vXFm64AsASAAvA5AIcbjbbjo3XpdRz2IEqlEqanp1EoFKCUQqFQcMboSyWNJSciomFwjDus2VngIx8Btm7tvu2aa4BNm1D9ylewYcMGLCwsdP2L7glenWPqUT0PERHFj2Pcg3r+eeADH3Ba1qtWtYX2g+edBxw86IxZf/zjwNKlKJVKvpO4eq0c1m+3N5f1JCLKJgZ3pxdfBC6+2AnrlSuB225bvOlPALwGTjf4L27bhupXv9p19zDLbXaG9GWXXdZ3tzeX9SQiyii/A7xNukS+AMtLL4lMTnouivJngLwmYIWxTr1WDvM7DWjYxw/7PEREZC/w7GAe5uaAjRudlvXy5c7hWq4ty5cjB6dl/UcAfuLzEH7d0kccccTi7/l8vm2Cl9dJRsRnnkFQtzcnkhERZdPSpAsQu/37gdWru6//8IeBz3wGOPJIXDIygjBT9jq7pb1WI5ufn2/7n37GoHt1e5dKJQY1EVHGZK/Ffe+9r/5+ySXACy84neKf/zxw5JEAwo0Tex3fHOZc1GHHoHn8NBEReclecL///a+OYE9PA697Xde/eC1EMjo6inw+H9gtHWamt9djd1qyZAm7vYmIyFP2gjukznHqL33pSzhw4MDi8qBeobpy5UrPx2oN6taxaT+NRoOhTUREnhjcHZrj1LOzs4vXdY5T9+vll19uO7SruUY41wYnIqJ+Mbg7hBmn9vPcc88FPm4nrg1ORET9YnB3GGZFsqCWstf9eUgXERH1i8HdYZgVyaampqB8zgDmd//OU2sytImIKAiDu8Mw3delUgmXXnppV3iz+5uIiHRhcHcYtvv6xhtvxG233cbubyIiigRP66lJtVpFuVzGzMwMxsfHMTU1xbAmIqKBBJ3WM3tLnkagc6nT5tm9ADC8iYhIK3aV++jn/NjDHEJGRETUD7a4PQS1oAF0dYkPcwgZERFRPzjG3aFarWLDhg1YWFjoui2fz2N+fr6tdZ3L5XDEEUe0rbTWVCgUUKvVoiwuERGlUNAYN7vKXdVqFatWrcJFF13kGdoAMDs769klDoAroBERUSwY3PBen7wfs7OzXAGNiIhiwa5yAMViEfV6PfB/crkc5ufn4VVfS5YsweHDh6MqHhERZYxxXeVKqZpS6rtKqQeUUokfoN1rElnz/Nh+Ozl+XetERES6JdlV/ssi8ha/PYo4Ba1DnsvlcMstt6BUKvmehjPo3NpEREQ6cYwb3uuTA4BSavF47Gq1ytNwEhFR4pIKbgHwz0qpXUqpCa9/UEpNKKV2KqV2Pvvss5EWpnN98nw+j9HR0cWu8dbjuDkJjYiIkpTI5DSl1HEislcp9QYA2wF8RET+ze//416r3G+yGo/LJiKiOBg3OU1E9ro/9wP4ewC/kEQ5/HAlNCIiMlXswa2UWqaUWt78HcCvAXgo7nIE8ZusFjSJjYiIKA5JtLhXA/h3pdSDAO4H8I8i8s0EyuGLk9CIiMhUsZ9kRES+D+Dn4n7efjQnm/H82kREZBqunEZERGQY4yanERER0WAY3ERERBZhcBMREVmEwU1ERGQRBjcREZFFGNxEREQWYXATERFZhMFNRERkkcwHd7VaRbFYxMjICIrFIqrVatJFIiIi8hX7kqcmqVarmJiYwNzcHID2825zeVMiIjJRplvc5XJ5MbSb5ubmUC6XEyoRERFRsEwHN8+7TUREtsl0cPO820REZJtMBzfPu01ERLbJdHCXSiVMT0+jUChAKYVCoYDp6WlOTCMiImPxfNxERESG4fm4iYiIUoLBTUREZBEGNxERkUUY3ERERBZhcBMREVmEwU1ERGQRBjcREZFFMhXcPIUnERHZLjOn9eQpPImIKA0y0+LmKTyJiCgNMhPcPIUnERGlQWaCm6fwJCKiNMhMcPMUnkRElAaZCW6ewpOIiNKAp/UkIiIyDE/rSURElBIMbiIiIoswuImIiCzC4CYiIrIIg5uIiMgiDG4iIiKLMLiJiIgswuAmIiKyiBULsCilngVQ1/RwqwAc0PRYacE66cY6acf66MY6acf66DZMnRRE5GivG6wIbp2UUjv9VqPJKtZJN9ZJO9ZHN9ZJO9ZHt6jqhF3lREREFmFwExERWSSLwT2ddAEMxDrpxjppx/roxjppx/roFkmdZG6Mm4iIyGZZbHETERFZK1PBrZQ6Wyn1qFLqCaXUlUmXJwlKqZpS6rtKqQeUUjvd61YqpbYrpR53f65IupxRUkrdrJTar5R6qOU6zzpQjr9yPzN7lFKnJVfy6PjUyVVKqb3uZ+UBpdS7W277uFsnjyql3pVMqaOjlDpBKfUtpdTDSqnvKaUud6/P7OckoE4y+TlRSr1WKXW/UupBtz6udq9fo5Ta4b7uryilxtzrX+P+/YR7e3HgJxeRTFwALAHwJIATAYwBeBDAyUmXK4F6qAFY1XHdZwFc6f5+JYA/S7qcEdfBOgCnAXioVx0AeDeAbQAUgDMA7Ei6/DHWyVUArvD435Pd789rAKxxv1dLkn4NmuvjGACnub8vB/CY+7oz+zkJqJNMfk7c9/pI9/dRADvc9/6rAC5wr/8CgEn398sAfMH9/QIAXxn0ubPU4v4FAE+IyPdF5CCA2wG8N+EymeK9AG5xf78FwPuSK0r0ROTfADzXcbVfHbwXwK3i+C8ARymljomloDHyqRM/7wVwu4j8RESeAvAEnO9XaojIPhHZ7f7+EoBHAByHDH9OAurET6o/J+57/WP3z1H3IgDOAnCHe33nZ6T52bkDwHqllBrkubMU3McB+H8tfz+N4A9dWgmAf1ZK7VJKTbjXrRaRfe7vzwBYnUzREuVXB1n/3Pwft+v35pYhlEzViduleSqcFhU/J+iqEyCjnxOl1BKl1AMA9gPYDqdX4Ucictj9l9bXvFgf7u0vAMgP8rxZCm5y/JKInAbgHAAfVkqta71RnH6cTB9qwDpYdBOAnwLwFgD7APxFoqVJgFLqSAB/C2CjiLzYeltWPycedZLZz4mILIjIWwAcD6c34afjeN4sBfdeACe0/H28e12miMhe9+d+AH8P58P2w2a3nvtzf3IlTIxfHWT2cyMiP3Q3TA0AX8Sr3ZyZqBOl1CicgKqKyN+5V2f6c+JVJ1n/nACAiPwIwLcAvA3OMMlS96bW17xYH+7trwcwO8jzZSm4/xvAG90Zf2NwJgfclXCZYqWUWqaUWt78HcCvAXgITj1scP9tA4A7kylhovzq4C4AH3BnDZ8B4IWWrtJU6xij/Q04nxXAqZML3FmyawC8EcD9cZcvSu7Y4xYAj4jIX7bclNnPiV+dZPVzopQ6Wil1lPv7EQB+Fc64/7cAnO/+W+dnpPnZOR/Av7q9Nv1LemZenBc4Mz8fgzMOUU66PAm8/hPhzPJ8EMD3mnUAZ5zlXwA8DuBuACuTLmvE9bAVTpfeIThjUB/0qwM4M0dvcD8z3wWwNunyx1gnt7mveY+70Tmm5f/Lbp08CuCcpMsfQX38Epxu8D0AHnAv787y5ySgTjL5OQHwswC+477uhwB80r3+RDg7KE8A+BqA17jXv9b9+wn39hMHfW6unEZERGSRLHWVExERWY/BTUREZBEGNxERkUUY3ERERBZhcBMREVmEwU1EUErlW87u9EzL2Z6k86xOSqmNSqmbkiorUdYxuIkIIjIrIm8RZ/nGLwC41v39Q3AWK2p1AZzjvokoAQxuIgpyB4BfbzmncBHAsQC+nWShiLKMwU1EvkTkOTirPJ3jXnUBgK8KV24iSgyDm4h62YpXu8vZTU6UMAY3EfVyJ4D1SqnTAOREZFfSBSLKMgY3EQUSkR/DOePRzWBrmyhxDG4iCmMrgJ8Dg5socTw7GBERkUXY4iYiIrIIg5uIiMgiDG4iIiKLMLiJiIgswuAmIiKyCIObiIjIIgxuIiIiizC4iYiILPL/AczYEjvtfjhPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 576x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig, ax = plt.subplots(1,1,figsize=(8,5))\n",
    "ax.plot(X, Y, 'o', color='black')\n",
    "ax.set_xlabel('TV')\n",
    "ax.set_ylabel('Sales')\n",
    "\n",
    "ax.plot(X, m_sklearn[0][0]*X+b_sklearn[0], color='red')\n",
    "ax.plot(X_pred, Y_pred_sklearn, 'o', color='blue')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='3'></a>\n",
    "## 3 - Linear Regression using Gradient Descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Functions to fit the models automatically are convenient to use, but for an in-depth understanding of the model and the maths behind it is good to implement an algorithm by yourself. Let's try to find linear regression coefficients $m$ and $b$, by minimising the difference between original values $y^{(i)}$ and predicted values $\\hat{y}^{(i)}$ with the **loss function** $L\\left(w, b\\right)  = \\frac{1}{2}\\left(\\hat{y}^{(i)} - y^{(i)}\\right)^2$ for each of the training examples. Division by $2$ is taken just for scaling purposes, you will see the reason below, calculating partial derivatives.\n",
    "\n",
    "To compare the resulting vector of the predictions $\\hat{Y}$ with the vector $Y$ of original values $y^{(i)}$, you can take an average of the loss function values for each of the training examples:\n",
    "\n",
    "$$E\\left(m, b\\right) = \\frac{1}{2n}\\sum_{i=1}^{n} \\left(\\hat{y}^{(i)} - y^{(i)}\\right)^2 = \n",
    "\\frac{1}{2n}\\sum_{i=1}^{n} \\left(mx^{(i)}+b - y^{(i)}\\right)^2,\\tag{1}$$\n",
    "\n",
    "where $n$ is a number of data points. This function is called the sum of squares **cost function**. To use gradient descent algorithm, calculate partial derivatives as:\n",
    "\n",
    "\\begin{align}\n",
    "\\frac{\\partial E }{ \\partial m } &= \n",
    "\\frac{1}{n}\\sum_{i=1}^{n} \\left(mx^{(i)}+b - y^{(i)}\\right)x^{(i)},\\\\\n",
    "\\frac{\\partial E }{ \\partial b } &= \n",
    "\\frac{1}{n}\\sum_{i=1}^{n} \\left(mx^{(i)}+b - y^{(i)}\\right),\n",
    "\\tag{2}\\end{align}\n",
    "\n",
    "and update the parameters iteratively using the expressions\n",
    "\n",
    "\\begin{align}\n",
    "m &= m - \\alpha \\frac{\\partial E }{ \\partial m },\\\\\n",
    "b &= b - \\alpha \\frac{\\partial E }{ \\partial b },\n",
    "\\tag{3}\\end{align}\n",
    "\n",
    "where $\\alpha$ is the learning rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Original arrays `X` and `Y` have different units. To make gradient descent algorithm efficient, you need to bring them to the same units. A common approach to it is called **normalization**: substract the mean value of the array from each of the elements in the array and divide them by standard deviation (a statistical measure of the amount of dispersion of a set of values). If you are not familiar with mean and standard deviation, do not worry about this for now - this is covered in the next Course of Specialization.\n",
    "\n",
    "Normalization is not compulsory - gradient descent would work without it. But due to different units of `X` and `Y`, the cost function will be much steeper. Then you would need to take a significantly smaller learning rate $\\alpha$, and the algorithm will require thousands of iterations to converge instead of a few dozens. Normalization helps to increase the efficiency of the gradient descent algorithm.\n",
    "\n",
    "Normalization is implemented in the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "X_norm = (X - np.mean(X))/np.std(X)\n",
    "Y_norm = (Y - np.mean(Y))/np.std(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define cost function according to the equation $(1)$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "def E(m, b, X, Y):\n",
    "    return 1/(2*len(Y))*np.sum((m*X + b - Y)**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex05'></a>\n",
    "### Exercise 5\n",
    "\n",
    "\n",
    "Define functions `dEdm` and `dEdb` to calculate partial derivatives according to the equations $(2)$. This can be done using vector form of the input data `X` and `Y`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "def dEdm(m, b, X, Y):\n",
    "    ### START CODE HERE ### (~ 1 line of code)\n",
    "    # Use the following line as a hint, replacing all None.\n",
    "    res = 1/len(Y)*np.dot(m*X + b - Y, X)\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return res\n",
    "    \n",
    "\n",
    "def dEdb(m, b, X, Y):\n",
    "    ### START CODE HERE ### (~ 1 line of code)\n",
    "    # Replace None writing the required expression fully.\n",
    "    res = 1/len(Y)*np.sum(m*X + b - Y)\n",
    "    ### END CODE HERE ###\n",
    "    \n",
    "    return res\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-0.7822244248616067\n",
      "5.151434834260726e-16\n",
      "0.21777557513839355\n",
      "5.000000000000001\n"
     ]
    }
   ],
   "source": [
    "print(dEdm(0, 0, X_norm, Y_norm))\n",
    "print(dEdb(0, 0, X_norm, Y_norm))\n",
    "print(dEdm(1, 5, X_norm, Y_norm))\n",
    "print(dEdb(1, 5, X_norm, Y_norm))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### __Expected Output__ \n",
    "\n",
    "```Python\n",
    "-0.7822244248616067\n",
    "5.098005351200641e-16\n",
    "0.21777557513839355\n",
    "5.000000000000002\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "w2_unittest.test_partial_derivatives(dEdm, dEdb, X_norm, Y_norm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='ex06'></a>\n",
    "### Exercise 6\n",
    "\n",
    "\n",
    "Implement gradient descent using expressions $(3)$:\n",
    "\\begin{align}\n",
    "m &= m - \\alpha \\frac{\\partial E }{ \\partial m },\\\\\n",
    "b &= b - \\alpha \\frac{\\partial E }{ \\partial b },\n",
    "\\end{align}\n",
    "\n",
    "where $\\alpha$ is the `learning_rate`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": [
    "def gradient_descent(dEdm, dEdb, m, b, X, Y, learning_rate = 0.001, num_iterations = 1000, print_cost=False):\n",
    "    for iteration in range(num_iterations):\n",
    "        ### START CODE HERE ### (~ 2 lines of code)\n",
    "        m_new = m - learning_rate * dEdm(m, b, X, Y)\n",
    "        b_new = b - learning_rate * dEdb(m, b, X, Y)\n",
    "        ### END CODE HERE ###\n",
    "        m = m_new\n",
    "        b = b_new\n",
    "        if print_cost:\n",
    "            print (f\"Cost after iteration {iteration}: {E(m, b, X, Y)}\")\n",
    "        \n",
    "    return m, b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(0.49460408269589495, -3.4915181856831644e-16)\n",
      "(0.9791767513915026, 4.521910375044022)\n"
     ]
    }
   ],
   "source": [
    "print(gradient_descent(dEdm, dEdb, 0, 0, X_norm, Y_norm))\n",
    "print(gradient_descent(dEdm, dEdb, 1, 5, X_norm, Y_norm, learning_rate = 0.01, num_iterations = 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### __Expected Output__ \n",
    "\n",
    "```Python\n",
    "(0.49460408269589495, -3.489285249624889e-16)\n",
    "(0.9791767513915026, 4.521910375044022)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m All tests passed\n"
     ]
    }
   ],
   "source": [
    "w2_unittest.test_gradient_descent(gradient_descent, dEdm, dEdb, X_norm, Y_norm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now run the gradient descent method starting from the initial point $\\left(m_0, b_0\\right)=\\left(0, 0\\right)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cost after iteration 0: 0.20629997559196597\n",
      "Cost after iteration 1: 0.19455197461564464\n",
      "Cost after iteration 2: 0.19408205457659178\n",
      "Cost after iteration 3: 0.19406325777502967\n",
      "Cost after iteration 4: 0.19406250590296714\n",
      "Cost after iteration 5: 0.19406247582808467\n",
      "Cost after iteration 6: 0.19406247462508938\n",
      "Cost after iteration 7: 0.19406247457696957\n",
      "Cost after iteration 8: 0.19406247457504477\n",
      "Cost after iteration 9: 0.19406247457496775\n",
      "Cost after iteration 10: 0.1940624745749647\n",
      "Cost after iteration 11: 0.19406247457496456\n",
      "Cost after iteration 12: 0.19406247457496456\n",
      "Cost after iteration 13: 0.19406247457496456\n",
      "Cost after iteration 14: 0.19406247457496456\n",
      "Cost after iteration 15: 0.19406247457496456\n",
      "Cost after iteration 16: 0.19406247457496456\n",
      "Cost after iteration 17: 0.19406247457496456\n",
      "Cost after iteration 18: 0.19406247457496456\n",
      "Cost after iteration 19: 0.19406247457496456\n",
      "Cost after iteration 20: 0.19406247457496456\n",
      "Cost after iteration 21: 0.19406247457496456\n",
      "Cost after iteration 22: 0.19406247457496456\n",
      "Cost after iteration 23: 0.19406247457496456\n",
      "Cost after iteration 24: 0.19406247457496456\n",
      "Cost after iteration 25: 0.19406247457496456\n",
      "Cost after iteration 26: 0.19406247457496456\n",
      "Cost after iteration 27: 0.19406247457496456\n",
      "Cost after iteration 28: 0.19406247457496456\n",
      "Cost after iteration 29: 0.19406247457496456\n",
      "Gradient descent result: m_min, b_min = 0.7822244248616068, -6.075140390748858e-16\n"
     ]
    }
   ],
   "source": [
    "m_initial = 0; b_initial = 0; num_iterations = 30; learning_rate = 1.2\n",
    "m_gd, b_gd = gradient_descent(dEdm, dEdb, m_initial, b_initial, \n",
    "                              X_norm, Y_norm, learning_rate, num_iterations, print_cost=True)\n",
    "\n",
    "print(f\"Gradient descent result: m_min, b_min = {m_gd}, {b_gd}\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember, that the initial datasets were normalized. To make the predictions, you need to normalize `X_pred` array, calculate `Y_pred` with the linear regression coefficients `m_gd`, `b_gd` and then **denormalize** the result (perform the reverse process of normalization):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TV marketing expenses:\n",
      "[ 50 120 280]\n",
      "Predictions of sales using Scikit_Learn linear regression:\n",
      "[[ 9.40942557 12.7369904  20.34285287]]\n",
      "Predictions of sales using Gradient Descent:\n",
      "[ 9.40942557 12.7369904  20.34285287]\n"
     ]
    }
   ],
   "source": [
    "X_pred = np.array([50, 120, 280])\n",
    "# Use the same mean and standard deviation of the original training array X\n",
    "X_pred_norm = (X_pred - np.mean(X))/np.std(X)\n",
    "Y_pred_gd_norm = m_gd * X_pred_norm + b_gd\n",
    "# Use the same mean and standard deviation of the original training array Y\n",
    "Y_pred_gd = Y_pred_gd_norm * np.std(Y) + np.mean(Y)\n",
    "\n",
    "print(f\"TV marketing expenses:\\n{X_pred}\")\n",
    "print(f\"Predictions of sales using Scikit_Learn linear regression:\\n{Y_pred_sklearn.T}\")\n",
    "print(f\"Predictions of sales using Gradient Descent:\\n{Y_pred_gd}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should have gotten similar results as in the previous sections. \n",
    "\n",
    "Well done! Now you know how gradient descent algorithm can be applied to train a real model. Re-producing results manually for a simple case should give you extra confidence that you understand what happends under the hood of commonly used functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "graded"
    ]
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "C1_W1_Assignment_Solution.ipynb",
   "provenance": []
  },
  "coursera": {
   "schema_names": [
    "AI4MC1-1"
   ]
  },
  "grader_version": "1",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "vscode": {
   "interpreter": {
    "hash": "478841ab876a4250505273c8a697bbc1b6b194054b009c227dc606f17fb56272"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
